Communication terminal, method for controlling display of image, and non-transitory computer-readable storage medium

ABSTRACT

A communication terminal includes a memory, a receiver, and circuitry. The memory stores image data identification information identifying image data in association with image type information indicating a type of an image represented by the image data. The receiver receives specific image data and specific image data identification information identifying the specific image data from another communication terminal. The circuitry controls a display to display a specific image represented by the specific image data received by the receiver in a size according to the image type information associated with the specific image identification information in the memory.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. § 119(a) to Japanese Patent Application Nos. 2016-193223, filedon Sep. 30, 2016 and 2017-175142, filed on Sep. 12, 2017 in the JapanPatent Office, the entire disclosures of which are hereby incorporatedby reference herein.

BACKGROUND Technical Field

The present disclosure relates to a communication terminal, a method forcontrolling display of image, and a non-transitory computer-readablestorage medium.

Description of the Related Art

Video conference systems are now in widespread use, allowing users atremote places to hold a meeting via a communication network such as theInternet. In such video conference systems, a communication terminal fora remote conference system is provided in a conference room whereattendants of one party in a remote conference are attending. Thiscommunication terminal collects an image or video of the conference roomincluding the attendants and sound such as speech made by theattendants, and transmits digital data converted from the collectedimage (video) and/or sound to the other party's terminal provided at adifferent conference room. Based on the transmitted digital data, theother party's terminal displays images on a display or outputs audiofrom a speaker in the different conference room to enable video calling.This enables to carry out a conference among remote sites, in a stateclose to an actual conference.

On the other hand, a technique is known that connects, to acommunication terminal, an image capturing device that is capable ofcapturing a full spherical panoramic image in real time, and distributesthe full spherical panoramic image from the image capturing device toeach communication terminal of the other party. Each communicationterminal sequentially converts the received full spherical panoramicimage to a rectangular flat image, and displays the flat image on adisplay or the like.

SUMMARY

A communication terminal includes a memory, a receiver, and circuitry.The memory stores image data identification information identifyingimage data in association with image type information indicating a typeof an image represented by the image data. The receiver receivesspecific image data and specific image data identification informationidentifying the specific image data from another communication terminal.The circuitry controls a display to display a specific image representedby the specific image data received by the receiver in a size accordingto the image type information associated with the specific imageidentification information in the memory.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the embodiments and many of theattendant advantages and features thereof can be readily obtained andunderstood from the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1A is a left side view of an image capturing device according to anembodiment of the present disclosure;

FIG. 1B is a front view of the image capturing device of FIG. 1A;

FIG. 1C is a plan view of the image capturing device of FIG. 1A;

FIG. 2 illustrates how a user uses the image capturing device, accordingto an embodiment of the present disclosure;

FIG. 3A is a view illustrating a front side of a hemispherical imagecaptured by the image capturing device according to an embodiment of thepresent disclosure;

FIG. 3B is a view illustrating a back side of the hemispherical imagecaptured by the image capturing device according to an embodiment of thepresent disclosure;

FIG. 3C is a view illustrating an image captured by the image capturingdevice represented by Mercator projection according to an embodiment ofthe present disclosure;

FIG. 4A illustrates how the image represented by Mercator projectioncovers a surface of a sphere according to an embodiment of the presentdisclosure;

FIG. 4B is a view illustrating a full spherical panoramic imageaccording to an embodiment of the present disclosure;

FIG. 5 is a view illustrating positions of a virtual camera and apredetermined area in a case in which the full spherical panoramic imageis represented as a three-dimensional solid sphere according to anembodiment of the present disclosure;

FIG. 6A is a perspective view of FIG. 5;

FIG. 6B is a view illustrating an image of the predetermined area on adisplay of a communication terminal according to an embodiment of thepresent disclosure;

FIG. 7 is a view illustrating a relation between predetermined-areainformation and a predetermined-area image according to an embodiment ofthe present disclosure;

FIG. 8 is a schematic view illustrating an image communication systemaccording to an embodiment of the present disclosure;

FIG. 9 is a schematic block diagram illustrating a hardwareconfiguration of the image capturing device according to an embodimentof the present disclosure;

FIG. 10 is a schematic block diagram illustrating a hardwareconfiguration of a videoconference terminal, according to an embodimentof the present disclosure;

FIG. 11 is a schematic block diagram illustrating a hardwareconfiguration of any one of a communication management system and apersonal computer (PC), according to an embodiment of the presentdisclosure;

FIG. 12 is a schematic block diagram illustrating a hardwareconfiguration of a smartphone, according to an embodiment of the presentdisclosure;

FIGS. 13A and 13B are a schematic block diagram illustrating afunctional configuration of the image communication system according toan embodiment of the present disclosure;

FIG. 14 is a conceptual diagram illustrating an image type managementtable, according to an embodiment of the present disclosure;

FIG. 15 is a conceptual diagram illustrating an image capturing devicemanagement table, according to an embodiment of the present disclosure;

FIG. 16 is a conceptual diagram illustrating a session management table,according to an embodiment of the present disclosure;

FIG. 17 is a conceptual diagram illustrating an image type managementtable, according to an embodiment of the present disclosure;

FIG. 18 is a sequence diagram illustrating operation of participating ina specific communication session according to an embodiment of thepresent disclosure;

FIG. 19 is a view illustrating a selection screen for acceptingselection of a desired communication session (virtual conference),according to an embodiment of the present disclosure;

FIG. 20 is a sequence diagram illustrating an operation of managingimage type information, according to an embodiment of the presentdisclosure;

FIG. 21 is a sequence diagram illustrating an image data transmissionprocess in video calling, according to an embodiment of the presentdisclosure;

FIG. 22A illustrates an example state of video calling in a case whichthe image capturing device of FIGS. 1A to 1C is not used, according toan embodiment of the present disclosure;

FIG. 22B illustrates an example state of video calling in a case whichthe image capturing device of FIGS. 1A to 1C is used, according to anembodiment of the present disclosure;

FIGS. 23A and 23B are views, each illustrating a display example at asite B according to an embodiment of the present disclosure;

FIGS. 24A and 24B are views, each illustrating another display exampleat the site B according to an embodiment of the present disclosure, and

FIGS. 25A and 25B are views, each illustrating still another displayexample at the site B according to an embodiment of the presentdisclosure.

The accompanying drawings are intended to depict embodiments of thepresent disclosure and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

As used herein, the singular forms “a”, “an”, and “the” are intended toinclude the multiple forms as well, unless the context clearly indicatesotherwise.

Referring to the drawings, an embodiment of the present disclosure isdescribed.

Overview of Embodiment

<Generation of Full Spherical Panoramic Image>

With reference to FIGS. 1 to 7, a description is given of generating afull spherical panoramic image.

First, a description is given of an external view of an image capturingdevice 1, with reference to FIGS. 1A to 1C. The image capturing device 1is a digital camera for capturing images from which a 360-degree fullspherical panoramic image is generated. FIGS. 1A to 1C are respectivelya left side view, a front view, and a plan view of the image capturingdevice 1.

As illustrated in FIG. 1A, the image capturing device 1 has a shape suchthat one can hold it with one hand. Further, as illustrated in FIGS. 1Ato 1C, an imaging element 103 a is provided on a front side (anteriorside) of an upper section of the image capturing device 1, and animaging element 103 b is provided on a back side (rear side) thereof.These imaging elements (image sensors) 103 a and 103 b are used incombination with optical members (e.g., fisheye lenses 102 a and 102 b,described later), each being capable of capturing a hemispherical imagehaving an angle of view of 180 degrees or wider. Furthermore, asillustrated in FIG. 1B, an operation unit 115 such as a shutter buttonis provided on an opposite side of the front side of the image capturingdevice 1.

Hereinafter, a description is given of a situation where the imagecapturing device 1 is used with reference to FIG. 2. FIG. 2 illustratesan example of how a user uses the image capturing device 1. Asillustrated in FIG. 2, for example, the image capturing device 1 is usedfor capturing objects surrounding the user who is holding the imagecapturing device 1 in his/her hand. In this case, the imaging elements103 a and 103 b illustrated in FIGS. 1A to 1C capture the objectssurrounding the user to obtain two hemispherical images.

Hereinafter, a description is given of an overview of an operation ofgenerating the full spherical panoramic image from the image captured bythe image capturing device 1, with reference to FIGS. 3A to 3C and FIGS.4A and 4B. FIG. 3A is a view illustrating a hemispherical image (frontside) captured by the image capturing device 1. FIG. 3B is a viewillustrating a hemispherical image (back side) captured by the imagecapturing device 1. FIG. 3C is a view illustrating an image representedby Mercator projection. The image represented by Mercator projection asillustrated in FIG. 3C is referred to as a “Mercator image” hereinafter.FIG. 4A illustrates an example of how the Mercator image covers asurface of a sphere. FIG. 4B is a view illustrating the full sphericalpanoramic image.

As illustrated in FIG. 3A, an image captured by the imaging element 103a is a curved hemispherical image (front side) taken through the fisheyelens 102 a described later. Also, as illustrated in FIG. 3B, an imagecaptured by the imaging element 103 b is a curved hemispherical image(back side) taken through the fisheye lens 102 b described later. Thehemispherical image (front side) and the hemispherical image (backside), which is reversed by 180-degree from each other, is combined bythe image capturing device 1. Thus, the Mercator image as illustrated inFIG. 3C is generated.

The Mercator image is pasted on the sphere surface using Open GraphicsLibrary for Embedded Systems (OpenGL ES) as illustrated in FIG. 4A.Thus, the full spherical panoramic image as illustrated in FIG. 4B isgenerated. In other words, the full spherical panoramic image isrepresented as the Mercator image facing toward a center of the sphere.It should be noted that OpenGL ES is a graphic library used forvisualizing two-dimensional (2D) and three-dimensional (3D) data. Thefull spherical panoramic image is either a still image or a movie.

One may feel strange viewing the full spherical panoramic image, becausethe full spherical panoramic image is an image attached to the spheresurface. To resolve this strange feeling, an image of a predeterminedarea, which is a part of the full spherical panoramic image, isdisplayed as a flat image having fewer curves. The image of thepredetermined area is referred to as a “predetermined-area image”hereinafter. Hereinafter, a description is given of displaying thepredetermined-area image with reference to FIG. 5 and FIGS. 6A and 6B.

FIG. 5 is a view illustrating positions of a virtual camera IC and apredetermined area T in a case in which the full spherical panoramicimage is represented as a three-dimensional solid sphere. The virtualcamera IC corresponds to a position of a point of view (viewpoint) of auser who is viewing the full spherical panoramic image represented asthe three-dimensional solid sphere. FIG. 6A is a perspective view ofFIG. 5. FIG. 6B is a view illustrating the predetermined-area imagedisplayed on a display. In FIG. 6A, the full spherical panoramic imageillustrated in FIG. 4B is represented as a three-dimensional solidsphere CS. Assuming that the generated full spherical panoramic image isthe solid sphere CS, the virtual camera IC is outside of the fullspherical panoramic image as illustrated in FIG. 5. The predeterminedarea T in the full spherical panoramic image is an imaging area of thevirtual camera IC. Specifically, the predetermined area T is specifiedby predetermined-area information indicating a position coordinate(x(rH), y(rV), angle of view α (angle)) including an angle of view ofthe virtual camera IC in a three-dimensional virtual space containingthe full spherical panoramic image. Zooming of the predetermined area Tis implemented by enlarging or reducing a range (arc) of the angle ofview α. Further, zooming of the predetermined area T is implemented bymoving the virtual camera IC toward or away from the full sphericalpanoramic image.

The predetermined-area image, which is an image of the predeterminedarea T illustrated in FIG. 6A, is displayed as an imaging area of thevirtual camera IC, as illustrated in FIG. 6B. FIG. 6B illustrates thepredetermined-area image represented by the predetermined-areainformation that is set by default. In another example, thepredetermined-area image may be specified by an imaging area (X, Y, Z)of the virtual camera IC, i.e., the predetermined area T, rather thanthe predetermined-area information, i.e., the position coordinate of thevirtual camera IC. A description is given hereinafter using the positioncoordinate (x(rH), y(rV), and an angle of view α (angle)) of the virtualcamera IC.

Hereinafter, a description is given of a relation between thepredetermined-area information and the predetermined area T withreference to FIG. 7. FIG. 7 is a view illustrating a relation betweenthe predetermined-area information and the predetermined area T. Asillustrated in FIG. 7, a center point CP of 2L provides the parameters(x, y) of the predetermined-area information, where 2L denotes adiagonal angle of view of the predetermined area T specified the angleof view α of the virtual camera IC. Distance f is a distance from thevirtual camera IC to the central point CP. L is a distance between thecenter point CP and a given vertex of the predetermined area T (2L is adiagonal line). In FIG. 7, a trigonometric function equation generallyexpressed by the following equation (1) is satisfied.

Lf=tan(α/2)  (Equation 1)

<Overview of Image Communication System>

Hereinafter, a description is given of an overview of a configuration ofan image communication system according to this embodiment withreference to FIG. 8. FIG. 8 is a schematic diagram illustrating aconfiguration of the image communication system according to thisembodiment.

As illustrated in FIG. 8, the image communication system according tothis embodiment includes an image capturing device 1 a, an imagecapturing device 1 b, a videoconference terminal 3, a communicationmanagement system 5, a persona computer (PC) 7, an image capturingdevice 8, and a smartphone 9. They communicate data with one another viaa communication network 100 such as the Internet. The communicationnetwork 100 may be either a wireless network or a wired network.

The image capturing device 1 a and the image capturing device 1 b areeach a special digital camera, which captures an image of a subject orsurroundings to obtain two hemispherical images, from which a fullspherical panoramic image is generated, as described above. By contrast,the image capturing device 8 is a general-purpose digital camera thatcaptures an image of a subject or surroundings to obtain a general flatimage.

The videoconference terminal 3 is a terminal dedicated tovideoconferencing. The videoconference terminal 3 displays an image ofvideo calling on a display 4 via a wired cable such as a universalserial bus (USB). The videoconference terminal 3 usually captures animage by a camera 312, which is described later. However, in a case inwhich the videoconference terminal 3 is connected to a cradle 2 a onwhich the image capturing device 1 a is mounted, the image capturingdevice 1 a is preferentially used. Accordingly, two hemispherical imagesare obtained, from which a full spherical panoramic image is generated.When a wired cable is used for connecting the videoconference terminal 3and the cradle 2 a, the cradle 2 a not only enables communicationsbetween the image capturing device 1 a and the videoconference terminal3 but also supplies power with the image capturing device 1 a and holdsthe image capturing device 1 a. In this disclosure, the image capturingdevice 1 a, the cradle 2 a, the videoconference terminal 3, and thedisplay 4 are located at the same site A. Further, in the site A, fourusers A1, A2, A3 and A4 are participating in video calling.

The communication management system 5 manages communication ofvideoconference terminal 3, the PC 7 and the smartphone 9. Further, thecommunication management system 5 manages types (a general image typeand a special image type) of image data exchanged. In this disclosure,the special image is a full spherical panoramic image. The communicationmanagement system 5 is located, for example, at a service provider thatprovides video communication service. The communication managementsystem 5 may be configured as a single computer or a plurality ofcomputers to which divided portions (functions, means, or storages) arearbitrarily allocated.

The PC 7 performs video calling with the image capturing device 8connected thereto. In this disclosure, the PC 7 and the image capturingdevice 8 are located at the same site C. At the site C, one user C isparticipating in video calling.

The smartphone 9 includes a display 917, which is described later, anddisplays an image of video calling on the display 917. The smartphone 9includes a complementary metal oxide semiconductor (CMOS) sensor 905,and usually captures an image with the CMOS sensor 905. In addition, thesmartphone 9 is also capable of obtaining data of two hemisphericalimages captured by the image capturing device 1 b, based on which thefull spherical panoramic image is generated, by wireless communicationsuch as Wireless Fidelity (Wi-Fi) and Bluetooth (registered trademark).In a case in which wireless communication is used for obtaining data oftwo hemispherical images, a cradle 2 b just supplies power with theimage capturing device 1 b and holds the image capturing device 1 b. Inthis disclosure, the image capturing device 1 b, the cradle 2 b, and thesmartphone 9 are located at the same site B. Further, in the site B, twousers B1 and B2 are participating in video calling.

The videoconference terminal 3, the PC 7 and the smartphone 9 are eachan example of a communication terminal. OpenGL ES is installed in eachof those communication terminals to enable each communication terminalto generate predetermined-area information that indicates a partial areaof a full spherical panoramic image, or to generate a predetermined-areaimage from a full spherical panoramic image that is transmitted from adifferent communication terminal.

The arrangement of the terminals, apparatuses and users illustrated inFIG. 8 is just an example, and any other suitable arrangement willsuffice. For example, in the site C, an image capturing device that iscapable of performing image capturing for a full spherical panoramicimage may be used in place of the image capturing device 8. In addition,examples of the communication terminal include a digital television, asmartwatch, and a car navigation device. Hereinafter, any arbitrary oneof the image capturing device 1 a and the image capturing device 1 b isreferred to as “the image capturing device 1”.

<Hardware Configuration According to Embodiment>

Hereinafter, a description is given of hardware configurations of theimage capturing device 1, the videoconference terminal 3, thecommunication management system 5, the PC 7, and the smartphone 9according to this embodiment with reference to FIGS. 9 to 12. Since theimage capturing device 8 is a general-purpose camera, a detaileddescription thereof is omitted.

<Hardware Configuration of Image Capturing Device 1>

First, a description is given of a hardware configuration of the imagecapturing device 1 with reference to FIG. 9. FIG. 9 is a block diagramillustrating a hardware configuration of the image capturing device 1. Adescription is given hereinafter of a case in which the image capturingdevice 1 is a full spherical (omnidirectional) image capturing devicehaving two imaging elements. However, the image capturing device 1 mayinclude any suitable number of imaging elements, providing that itincludes at least two imaging elements. In addition, the image capturingdevice 1 is not necessarily an image capturing device dedicated toomnidirectional image capturing. Alternatively, an externalomnidirectional image capturing unit may be attached to ageneral-purpose digital camera or a smartphone to implement an imagecapturing device having substantially the same function as that of theimage capturing device 1.

As illustrated in FIG. 9, the image capturing device 1 includes animaging unit 101, an image processing unit 104, an imaging control unit105, a microphone 108, an audio processing unit 109, a centralprocessing unit (CPU) 111, a read only memory (ROM) 112, a static randomaccess memory (SRAM) 113, a dynamic random access memory (DRAM) 114, anoperation unit 115, a network interface (I/F) 116, a communicationdevice 117, and an antenna 117 a.

The imaging unit 101 includes two wide-angle lenses (so-called fish-eyelenses) 102 a and 102 b, each having an angle of view of equal to orgreater than 180 degrees so as to form a hemispherical image. Theimaging unit 101 further includes the two imaging elements 103 a and 103b corresponding to the wide-angle lenses 102 a and 102 b respectively.The imaging elements 103 a and 103 b each includes an image sensor suchas a CMOS sensor and a charge-coupled device (CCD) sensor, a timinggeneration circuit, and a group of registers. The image sensor convertsan optical image formed by the fisheye lenses 102 a and 102 b intoelectric signals to output image data. The timing generation circuitgenerates horizontal or vertical synchronization signals, pixel clocksand the like for the image sensor. Various commands, parameters and thelike for operations of the imaging elements 103 a and 103 b are set inthe group of registers.

Each of the imaging elements 103 a and 103 b of the imaging unit 101 isconnected to the image processing unit 104 via a parallel I/F bus. Inaddition, each of the imaging elements 103 a and 103 b of the imagingunit 101 is connected to the imaging control unit 105 via a serial I/Fbus such as an I2C bus. The image processing unit 104 and the imagingcontrol unit 105 are each connected to the CPU 111 via a bus 110.Furthermore, the ROM 112, the SRAM 113, the DRAM 114, the operation unit115, the network I/F 116, the communication device 117, and theelectronic compass 118 are also connected to the bus 110.

The image processing unit 104 acquires image data from each of theimaging elements 103 a and 103 b via the parallel I/F bus and performspredetermined processing on each image data. Thereafter, the imageprocessing unit 104 combines these image data to generate data of theMercator image as illustrated in FIG. 3C.

The imaging control unit 105 usually functions as a master device whilethe imaging elements 103 a and 103 b each usually functions as a slavedevice. The imaging control unit 105 sets commands and the like in thegroup of registers of the imaging elements 103 a and 103 b via the I2Cbus. The imaging control unit 105 receives necessary commands and thelike from the CPU 111. Further, the imaging control unit 105 acquiresstatus data and the like of the group of registers of the imagingelements 103 a and 103 b via the I2C bus. The imaging control unit 105sends the acquired status data and the like to the CPU 111.

The imaging control unit 105 instructs the imaging elements 103 a and103 b to output the image data at a time when the shutter button of theoperation unit 115 is pressed. The image capturing device 1 may displaya preview image on a display (e.g., a display of the videoconferenceterminal 3) or may support displaying movie. In this case, the imagedata are continuously output from the imaging elements 103 a and 103 bat a predetermined frame rate (frames per minute).

Furthermore, the imaging control unit 105 operates in cooperation withthe CPU 111 to synchronize times when the imaging elements 103 a and 103b output the image data. It should be noted that although in thisembodiment, the image capturing device 1 does not include a display unit(display), the image capturing device 1 may include the display.

The microphone 108 converts sounds to audio data (signal). The audioprocessing unit 109 acquires the audio data from the microphone 108 viaan I/F bus and performs predetermined processing on the audio data.

The CPU 111 controls overall operation of the image capturing device 1and performs necessary processing. The ROM 112 stores various programsfor the CPU 111.

The SRAM 113 and the DRAM 114 each operates as a work memory to storeprograms loaded from the ROM 112 for execution by the CPU 111 or data incurrent processing. More specifically, the DRAM 114 stores image datacurrently processed by the image processing unit 104 and data of theMercator image on which processing has been performed.

The operation unit 115 collectively refers to various operation keys, apower switch, the shutter button, and a touch panel having functions ofboth displaying information and receiving input from a user, which maybe used in combination. The user operates the operation keys to inputvarious photographing modes or photographing conditions.

The network I/F 116 collectively refers to an interface circuit such asa USB I/F that allows the image capturing device 1 to communicate datawith an external media such as an SD card or an external personalcomputer. The network I/F 116 supports at least one of wired andwireless communications. The data of the Mercator image, which is storedin the DRAM 114, is stored in the external media via the network I/F 116or transmitted to the external device such as the videoconferenceterminal 3 via the network I/F 116, as needed.

The communication device 117 communicates data with an external devicesuch as the videoconference terminal 3 via the antenna 117 a of theimage capturing device 1 by near distance wireless communication such asWi-Fi and Near Field Communication (NFC). The communication device 117is also capable of transmitting the data of Mercator image to theexternal device such as the videoconference terminal 3.

The electronic compass 118 calculates an orientation and a tilt (rollangle) of the image capturing device 1 from the Earth's magnetism tooutput orientation and tilt information. This orientation and tiltinformation is an example of related information, which is metadatadescribed in compliance with Exif. This information is used for imageprocessing such as image correction of captured images. Further, therelated information also includes a date and time when the image iscaptured by the image capturing device 1, and a size of the image data.

<Hardware Configuration of Videoconference Terminal 3>

Hereinafter, a description is given of a hardware configuration of thevideoconference terminal 3 with reference to FIG. 10. FIG. 10 is a blockdiagram illustrating a hardware configuration of the videoconferenceterminal 3. As illustrated in FIG. 10, the videoconference terminal 3includes a CPU 301, a ROM 302, a RAM 303, a flash memory 304, a solidstate drive (SSD) 305, a medium I/F 307, an operation key 308, a powerswitch 309, a bus line 310, a network I/F 311, a camera 312, an imagingelement I/F 313, a microphone 314, a speaker 315, an audio input/outputinterface 316, a display I/F 317, an external device connection I/F 318,a near-distance communication circuit 319, and an antenna 319 a for thenear-distance communication circuit 319.

The CPU 301 controls overall operation of the videoconference terminal3. The ROM 302 stores a control program for operating the CPU 301 suchas an Initial Program Loader (IPL). The RAM 303 is used as a work areafor the CPU 301. The flash memory 304 stores various data such as acommunication control program, image data, and audio data. The SSD 305controls reading or writing of various data to and from the flash memory304 under control of the CPU 301. A hard disk drive (HDD) may be used inplace of the SSD 305. The medium I/F 307 controls reading or writing(storing) of data with respect to a recording medium 306 such as a flashmemory. The operation key 308 is operated by a user to input a userinstruction such as a user selection of a destination of thevideoconference terminal 3. The power switch 309 is a switch that turnson or off the power of the videoconference terminal 3.

The network I/F 311 enables communication of data with an externaldevice through the communication network 100 such as the Internet. Thecamera 312 is an example of an imaging device capable of capturing asubject under control of the CPU 301 to obtain image data, and isincorporated in the videoconference terminal 3. The imaging element I/F313 is a circuit that controls driving of the camera 312. The microphone314 is an example of an audio collecting device capable of inputtingaudio, and is incorporated in the videoconference terminal 3. The audioinput/output interface 316 is a circuit for controlling input and outputof audio signals between the microphone 314 and the speaker 315 undercontrol of the CPU 301. The display I/F 317 is a circuit fortransmitting image data to an external display 320 under control of theCPU 301. The external device connection I/F 318 is an interface circuitthat connects the videoconference terminal 3 to various externaldevices. The near-distance communication circuit 319 is a communicationcircuit that communicates in compliance with the NFC (registeredtrademark), the Bluetooth (registered trademark) and the like.

The bus line 310 may be an address bus or a data bus, which electricallyconnects various elements such as the CPU 301 illustrated in FIG. 10.

The display 4 is an example of a display unit, such as a liquid crystalor organic electroluminescence (EL) display that displays an image of asubject, an operation icon, or the like. The display 4 is connected tothe display I/F 317 by a cable 4 c. The cable 4 c may be an analog redgreen blue (RGB) (video graphic array (VGA)) signal cable, a componentvideo cable, a high-definition multimedia interface (HDMI) (registeredtrademark) signal cable, or a digital video interactive (DVI) signalcable.

The camera 312 includes a lens and a solid-state imaging element thatconverts an image (video) of a subject to electronic data throughphotoelectric conversion. As the solid-state imaging element, forexample, a CMOS sensor or a CCD sensor is used. The external deviceconnection I/F 318 is capable of connecting an external device such asan external camera, an external microphone, or an external speakerthrough a USB cable or the like. In a case in which an external camerais connected, the external camera is driven in preference to thebuilt-in camera 312 under control of the CPU 301. Similarly, in a casein which an external microphone is connected or an external speaker isconnected, the external microphone or the external speaker is driven inpreference to the built-in microphone 314 or the built-in speaker 315under control of the CPU 301.

The recording medium 306 is removable from the videoconference terminal3. In addition to the flash memory 304, any suitable nonvolatile memory,such as an electrically erasable and programmable ROM (EEPROM), may beused, provided that it reads or writes data under control of CPU 301.

<Hardware Configuration of Communication Management System 5 and PC 7>

Hereinafter, a description is given of hardware configurations of thecommunication management system 5 and the PC 7, with reference to FIG.11. FIG. 11 is a block diagram illustrating a hardware configuration ofany one of the communication management system 5 and the PC 7. In thisdisclosure, both the communication management system 5 and the PC 7 areimplemented by a computer. Therefore, a description is given of aconfiguration of the communication management system 5, and thedescription of a configuration of the PC 7 is omitted, having the sameor substantially the same configuration as that of the communicationmanagement system 5.

The communication management system 5 includes a CPU 501, a ROM 502, aRAM 503, an HD 504, a hard disc drive (HDD) 505, a media drive 507, adisplay 508, a network IN 509, a keyboard 511, a mouse 512, a compactdisc rewritable (CD-RW) drive 514, and a bus line 510. The CPU 501controls entire operation of the communication management system 5. TheROM 502 stores a control program for controlling the CPU 501 such as anIPL. The RAM 503 is used as a work area for the CPU 501. The HD 504stores various data such as programs for the communication managementsystem 5. The HDD 505 controls reading and writing of data from and tothe HD 504 under control of the CPU 501. The media drive 507 controlsreading and writing (storing) of data from and to a recording medium 506such as a flash memory. The display 508 displays various types ofinformation such as a cursor, menus, windows, characters, or images. Thenetwork I/F 509 enables communication of data with an external devicethrough the communication network 100. The keyboard 511 includes aplurality of keys to allow a user to input characters, numbers, andvarious types of instructions. The mouse 512 allows a user to input aninstruction for selecting and executing various functions, selecting atarget for processing, or moving the cursor. The CD-RW drive 514controls reading and writing of data from and to a CD-RW 513, which isone example of removable recording medium. The bus line 510 electricallyconnects those elements of the communication management system 5 to eachother as illustrated in FIG. 11. Examples of the bus line 510 include anaddress bus and a data bus.

<Hardware Configuration of Smartphone 9>

Hereinafter, a description is given of hardware of the smartphone 9 withreference to FIG. 12. FIG. 12 is a block diagram illustrating a hardwareconfiguration of the smartphone 9. As illustrated in FIG. 12, thesmartphone 9 includes a CPU 901, a ROM 902, a RAM 903, an EEPROM 904, aCMOS sensor 905, an acceleration and orientation sensor 906, a mediumI/F 908, and a global positioning system (GPS) receiver 909.

The CPU 901 controls overall operation of the smartphone 9. The ROM 902stores a program for controlling the CPU 901, such as IPL. The RAM 903is used as a work area for the CPU 901. The EEPROM 904 reads or writesvarious data such as a control program for the smartphone 9 undercontrol of the CPU 901. The CMOS sensor 905 captures an object undercontrol of the CPU 901 to obtain image data. The acceleration andorientation sensor 906 includes various sensors such as anelectromagnetic compass or gyrocompass for detecting geomagnetism, andan acceleration sensor. The medium I/F 908 controls reading or writingof data with respect to a recording medium 907 such as a flash memory.The GPS receiver 909 receives a GPS signal from a GPS satellite.

The smartphone 9 further includes a far-distance communication circuit911, a camera 912, an imaging element I/F 913, a microphone 914, aspeaker 915, an audio input/output interface 916, a display 917, anexternal device connection I/F 918, a near-distance communicationcircuit 919, an antenna 919 a for the near-distance communicationcircuit 919, and a touch panel 921.

The far-distance communication circuit 911 is a circuit thatcommunicates with other device through the communication network 100.The camera 912 is an example of imaging device capable of capturing asubject under control of the CPU 901 to obtain image data, and isincorporated in the smartphone 9. The imaging element I/F 913 is acircuit that controls driving of the camera 912. The microphone 914 isan example of audio collecting device capable of inputting audio, and isincorporated in the smartphone 9. The audio input/output interface 916is a circuit for controlling input and output of audio signals betweenthe microphone 914 and the speaker 915 under control of the CPU 901. Thedisplay 917 is an example of a display unit, such as a liquid crystal ororganic electro luminescence (EL) display that displays an image of asubject, an operation icon, or the like. The external device connectionI/F 918 is an interface circuit that connects the smartphone 9 tovarious external devices. The near-distance communication circuit 919 isa communication circuit that communicates in compliance with the NFC,the Bluetooth and the like. The touch panel 921 is an example of aninput device to operate a smartphone 9 by touching a screen of thedisplay 917.

The smartphone 9 further includes a bus line 910. Examples of the busline 910 include an address bus and a data bus, which electricallyconnects the elements such as the CPU 901.

It should be noted that a recording medium such as a CD-ROM or a harddisk storing any one of the above-described programs may be distributeddomestically or overseas as a program product.

<Functional Configuration According to Embodiment>

Hereinafter, a description is given of a functional configuration of theimage communication system according to this embodiment, with referenceto FIGS. 13A and 13B to FIG. 17. FIGS. 13A and 13B are a block diagramillustrating functional configurations of the image capturing devices 1a and 1 b, the videoconference terminal 3, the communication managementsystem 5, the PC 7, and the smartphone 9, which constitute a part of theimage communication system according this embodiment.

<Functional Configuration of Image Capturing Device 1 a>

As illustrated in FIG. 13A, the image capturing device 1 a includes anacceptance unit 12 a, an image capturing unit 13 a, an audio collectingunit 14 a, a communication unit 18 a, and a data storage/read unit 19 a.These units are functions that are implemented by or that are caused tofunction by operating any of the elements illustrated in FIG. 9 incooperation with the instructions of the CPU 111 according to the imagecapturing device control program expanded from the SRAM 113 to the DRAM114.

The image capturing device 1 a further includes a memory 1000 a, whichis implemented by the ROM 112, the SRAM 113, and the DRAM 114illustrated in FIG. 9. The memory 1000 a stores therein a globallyunique identifier (GUID) identifying the own device (i.e., the imagecapturing device 1 a).

(Each Functional Unit of Image Capturing Device 1 a)

Hereinafter, referring to FIG. 9 and FIGS. 13A and 13B, a detaileddescription is given of each functional unit of the image capturingdevice 1 a according to the embodiment.

The acceptance unit 12 a of the image capturing device 1 a isimplemented by the operation unit 115 illustrated in FIG. 9, whenoperating under control of the CPU 111. The acceptance unit 12 areceives an instruction input from the operation unit 115 according to auser operation.

The image capturing unit 13 a is implemented by the imaging unit 101,the image processing unit 104, and the imaging control unit 105,illustrated in FIG. 9, when operating under control of the CPU 111. Theimage capturing unit 13 a captures an image of a subject or surroundingsto obtain captured-image data.

The audio collecting unit 14 a is implemented by the microphone 108 andthe audio processing unit 109 illustrated in FIG. 9, when operatingunder control of the CPU 111. The audio collecting unit 14 a collectssounds around the image capturing device 1 a.

The communication unit 18 a, which is implemented by instructions of theCPU 111, communicates data with a communication unit 38 of thevideoconference terminal 3 using the near-distance wirelesscommunication technology in compliance with such as NFC, Bluetooth, orWi-Fi.

The data storage/read unit 19 a, which is implemented by instructions ofthe CPU 111 illustrated in FIG. 9, stores data or information in thememory 1000 a and read out data or information from the memory 1000 a.

(Each Functional Unit of Image Capturing Device 1 b)

The image capturing device 1 b includes an acceptance unit 12 b, animage capturing unit 13 b, an audio collecting unit 14 b, acommunication unit 18 b, and a data storage/read unit 19 b. Thesefunctional units of the image capturing device 1 b implement the similaror substantially the similar functions as those of the acceptance unit12 a, the image capturing unit 13 a, the audio collecting unit 14 a, thecommunication unit 18 a, and the data storage/read unit 19 a of theimage capturing device 1 a, respectively. Therefore, redundantdescriptions thereof are omitted below. The image capturing device 1 bfurther includes a memory 1000 b, which is implemented by the ROM 112,the SRAM 113, and the DRAM 114 illustrated in FIG. 9. The memory 1000 bstores therein a GUID identifying the own device (i.e., the imagecapturing device 1 b).

<Functional Configuration of Videoconference Terminal 3>

As illustrated in FIG. 13A, the videoconference terminal 3 includes adata exchange unit 31, an acceptance unit 32, an image and audioprocessor 33, a display control 34, a determination unit 35, a generator36, a communication unit 38, and a data storage/read unit 39. Theseunits are functions that are implemented by or that are caused tofunction by operating any of the elements illustrated in FIG. 10 incooperation with the instructions of the CPU 301 according to thecontrol program for the videoconference terminal 3, expanded from theflash memory 304 to the RAM 303.

The videoconference terminal 3 further includes a memory 3000, which isimplemented by the ROM 302, the RAM 303, and the flash memory 304illustrated in FIG. 10. The memory 3000 stores an image type managementDB 3001 and an image capturing device management DB 3002. The image typemanagement DB 3001 is implemented by an image type management tableillustrated in FIG. 14. The image capturing device management DB 3002 isimplemented by an image capturing device management table illustrated inFIG. 15.

(Image Type Management Table)

FIG. 14 is a conceptual diagram illustrating the image type managementtable according to an embodiment of this disclosure. The image typemanagement table stores an image data identifier (ID), an IP address ofa sender terminal, and a source name, in association with one another.The IP address is an example of a destination of a sender terminal. Theimage data ID is one example of image data identification informationfor identifying image data when performing video communication.Identical image data ID is assigned to image data transmitted from thesame sender terminal. Accordingly, a destination terminal (that is, acommunication terminal that receives image data) can identify a senderterminal from which the image data is received. The IP address of senderterminal is an IP address of a communication terminal that transmitsimage data identified by the image data ID that is associated with theIP address. The source name is a name for identifying an image capturingdevice that outputs the image data identified by the image data IDassociated with the source name. The source name is one example of imagetype information. This source name is generated by a communicationterminal such as the videoconference terminal 3 according to apredetermined naming rule.

The example of the image type management table illustrated in FIG. 14indicates that three communication terminals, whose IP addresses arerespectively “1.2.1.3”, “1.2.2.3”, and “1.3.1.3”, transmit image dataidentified by the image data ID “RS001”, “RS002”, and “RS003”,respectively. Further, according to the image type management tableillustrated in FIG. 14, the image types represented by the source namesof those three communication terminals are “Video_Theta”, “Video”, and“Video_Theta”, which indicate that the image types are the “specialimage”, “general image”, and “special image”, respectively. In thisdisclosure, the special image is a full spherical panoramic image.

In another example, data other than the image data may be stored in theimage type management table in association with image data ID. Examplesof the data other than the image data include audio data andpresentation material data shared on a screen.

(Image Capturing Device Management Table)

FIG. 15 is a conceptual diagram illustrating the image capturing devicemanagement table according to an embodiment. The image capturing devicemanagement table stores a vendor ID and a product ID from among theGUIDs of an image capturing device that is capable of obtaining twohemispherical images, from which a full spherical panoramic image isgenerated. As the GUID, a vendor ID (VID) and a product ID (PID) used ina USB device are used, for example. Those vendor ID and product ID arestored in a communication terminal such as a videoconference terminalbefore shipment. In another example, those IDs are added and storedafter shipment.

(Each Functional Unit of Videoconference Terminal 3)

Hereinafter, referring to FIG. 10 and FIGS. 13A and 13B, a detaileddescription is given of each functional unit of the videoconferenceterminal 3 according to the embodiment.

The data exchange unit 31 of the videoconference terminal 3 isimplemented by the network I/F 311 illustrated in FIG. 10, whenoperating under control of the CPU 301. The data exchange unit 31exchanges data or information with communication management system 5 viathe communication network 100.

The acceptance unit 32 is implemented by the operation key 308, whenoperating under control of the CPU 301. The acceptance unit 32 receivesselections or inputs from a user. An input device such as a touch panelmay be used as an alternative to or in place of the operation key 308.

The image and audio processor 33, which is implemented by instructionsof the CPU 301 illustrated in FIG. 10, processes image data that isobtained by capturing a subject by the camera 312. In addition, afterthe audio of the user is converted to an audio signal by the microphone314, the image and audio processor 33 processes audio data based on thisaudio signal.

Further, the image and audio processor 33 processes image data receivedfrom other communication terminals based on the image type informationsuch as the source name, to enable the display control 34 to control thedisplay 4 to display an image. More specifically, when the image typeinformation indicates a special image, the image and audio processor 33converts into full spherical image data to generate full sphericalpanoramic image data as illustrated in FIG. 4B based on the image datasuch as hemispherical image data as illustrated in FIGS. 3A and 3B, andfurther generates a predetermined-area image as illustrated in FIG. 6B.Furthermore, the image and audio processor 33 outputs, to the speaker315, an audio signal according to audio data that is received fromanother communication terminal via the communication management system5. The speaker 315 outputs audio based on the audio signal.

The display control 34 is implemented by the display I/F 317, whenoperating under control of the CPU 301. The display control 34 controlsthe display 4 to display images or characters.

The determination unit 35, which is implemented by instructions of theCPU 301, determines an image type according to the image data receivedfrom such as the image capturing device 1 a.

The generator 36 is implemented by instructions of the CPU 301. Thegenerator 36 generates the source name, which is one example of theimage type information, according to the above-described naming rule,based on a determination result determined by the determination unit 35indicating a general image or a special image (that is, full sphericalpanoramic image in this disclosure). For example, when the determinationunit 35 determines the image type as a general image, the generator 36generates the source name “Video” indicating a general image. Bycontrast, when the determination unit 35 determines the image type as aspecial image, the generator 36 generates the source name “Video Theta”indicating a special image.

The communication unit 38 is implemented by the near-distancecommunication circuit 319 and the antenna 319 a, when operating undercontrol of the CPU 301. The communication unit 38 communicates with thecommunication unit 18 a of the image capturing device 1 a using thenear-distance communication technology in compliance with such as NFC,Bluetooth, or Wi-Fi. Although in the above description the communicationunit 38 and the data exchange unit 31 have separate communication units,alternatively a shared communication unit may be used.

The data storage/read unit 39, which is implemented by instructions ofthe CPU 301 illustrated in FIG. 10, stores data or information in thememory 3000 and reads out data or information from the memory 3000.

<Functional Configuration of Communication Management System 5>

Hereinafter, referring to FIG. 11 and FIGS. 13A and 13B, a detaileddescription is given of each functional unit of the communicationmanagement system 5. The communication management system 5 includes adata exchange unit 51, a determination unit 55, a generator 56, and adata storage/read unit 59. These units are functions that areimplemented by or that are caused to function by operating any of theelements illustrated in FIG. 11 in cooperation with the instructions ofthe CPU 501 according to the control program for the communicationmanagement system 5, expanded from the HD 504 to the RAM 503.

The communication management system 5 further includes a memory 5000,which is implemented by the RAM 503 and the HD 504 illustrated in FIG.11. The memory 5000 stores a session management DB 5001 and an imagetype management DB 5002. The session management DB 5001 is implementedby a session management table illustrated in FIG. 16. The image typemanagement DB 5002 is implemented by an image type management tableillustrated in FIG. 17.

(Session Management Table)

FIG. 16 is a conceptual diagram illustrating the session managementtable according to an embodiment. The session management table stores asession ID and an IP address of participating communication terminal, inassociation with each other. The session ID is one example of sessionidentification information for identifying a session that implementsvideo calling. The session ID is generated for each virtual conferenceroom. The session ID is also stored in each communication terminal suchas the videoconference terminal 3. Each communication terminal selects adesired session ID from the session ID or IDs stored therein. The IPaddress of participating communication terminal indicates an IP addressof the communication terminal participating in a virtual conference roomidentified by the associated session ID.

(Image Type Management Table)

FIG. 17 is a conceptual diagram illustrating the image type managementtable according to an embodiment. The image type management tableillustrated in FIG. 17 stores, in addition to the information itemsstored in the image type management table illustrated in FIG. 14, thesame session ID as the session ID stored in the session managementtable, in association with one another. The example of the image typemanagement table illustrated in FIG. 17 indicates that threecommunication terminals whose IP addresses are respectively “1.2.1.3”,“1.2.2.3”, and “1.3.1.3” are participating in the virtual conferenceroom identified by the session ID “se101”. The communication managementsystem 5 stores the same image data ID, IP address of the senderterminal, and image type information as those stored in a communicationterminal such as the videoconference terminal 3 in order to transmitsuch information as the image type information to both a communicationterminal that is already in video calling and a newly participatingcommunication terminal that enters the virtual conference room after thevideo calling has started. Accordingly, the communication terminal thatis already in the video calling and the newly participatingcommunication terminal do not have to exchange such information as theimage type information with each other.

(Each Functional Unit of Communication Management System 5)

Hereinafter, referring to FIG. 11 and FIGS. 13A and 13B, a detaileddescription is given of each functional unit of the communicationmanagement system 5.

The data exchange unit 51 of the communication management system 5 isimplemented by the network I/F 509 illustrated in FIG. 11, whenoperating under control of the CPU 501. The data exchange unit 51exchanges data or information with the videoconference terminal 3 or thePC 7 via the communication network 100.

The determination unit 55, which is implemented by instructions of theCPU 501, performs various determinations.

The generator 56, which is implemented by instructions of the CPU 501,generates the image data ID.

The data storage/read unit 59 is implemented by the HDD 505 illustratedin FIG. 11, when operating under control of the CPU 501. The datastorage/read unit 59 stores data or information in the memory 5000 andread out data or information from the memory 5000.

<Functional Configuration of PC 7>

Hereinafter, referring to FIG. 11 and FIGS. 13A and 13B, a detaileddescription is given of a functional configuration of the PC 7. The PC 7has the same or substantially the same functions as those of thevideoconference terminal 3. In other words, as illustrated in FIG. 13B,the PC 7 includes a data exchange unit 71, an acceptance unit 72, animage and audio processor 73, a display control 74, a determination unit75, a generator 76, a communication unit 78, and a data storage/readunit 79. These units are functions that are implemented by or that arecaused to function by operating any of the hardware elements illustratedin FIG. 11 in cooperation with the instructions of the CPU 501 accordingto the control program for the PC 7, expanded from the HD 504 to the RAM503.

The PC 7 further includes a memory 7000, which is implemented by the ROM502, the RAM 503 and the HD 504 illustrated in FIG. 11. The memory 7000stores an image type management DB 7001 and an image capturing devicemanagement DB 7002. The image type management DB 7001 and the imagecapturing device management DB 7002 have the same configurations asthose of the image type management DB 3001 and the image capturingdevice management DB 3002, respectively. Therefore, redundantdescriptions thereof are omitted below.

(Each Functional Unit of PC 7)

The data exchange unit 71 of the PC 7, which is implemented by thenetwork I/F 509, when operating under control of the CPU 501 illustratedin FIG. 11, implements the similar or substantially the similar functionto that of the data exchange unit 31.

The acceptance unit 72, which is implemented by the keyboard 511 and themouse 512, when operating under control of the CPU 501, implements thesimilar or substantially the similar function to that of the acceptanceunit 32. The image and audio processor 73, which is implemented byinstructions of the CPU 501, implements the similar or substantially thesimilar function to that of the image and audio processor 33. Thedisplay control 74, which is implemented by instructions of the CPU 501,implements the similar or substantially the similar function to that ofthe display control 34. The determination unit 75, which is implementedby instructions of the CPU 501, implements the similar or substantiallythe similar function to that of the determination unit 35. The generator76, which is implemented by instructions of the CPU 501, implements thesimilar or substantially the similar function to that of the generator36. The communication unit 78, which is implemented by instructions ofthe CPU 501, implements the similar or substantially the similarfunction to that of the communication unit 38. The data storage/readunit 79, which is implemented by instructions of the CPU 501, storesdata or information in the memory 7000 and read out data or informationfrom the memory 7000.

<Functional Configuration of Smartphone 9>

Hereinafter, referring to FIG. 12 and FIGS. 13A and 13B, a detaileddescription is given of a functional configuration of the smartphone 9.The smartphone 9 has the same or substantially the same functions asthose of the videoconference terminal 3. In other words, as illustratedin FIG. 13B, the smartphone 9 includes a data exchange unit 91, anacceptance unit 92, an image and audio processor 93, a display control94, a determination unit 95, a generator 96, a communication unit 98,and a data storage/read unit 99. These units are functions that areimplemented by or that are caused to function by operating any of thehardware elements illustrated in FIG. 12 in cooperation with theinstructions of the CPU 901 according to the control program for thesmartphone 9 opened from the EEPROM 904 to the RAM 903.

The smartphone 9 further includes a memory 9000, which is implemented bythe ROM 902, the RAM 903, and the EEPROM 904 illustrated in FIG. 12. Thememory 9000 stores an image type management DB 9001 and an imagecapturing device management DB 9002. The image type management DB 9001and the image capturing device management DB 9002 have the sameconfigurations as those of the image type management DB 3001 and theimage capturing device management DB 3002, respectively. Therefore,redundant descriptions thereof are omitted below.

(Each Functional Unit of Smartphone 9)

The data exchange unit 91 of the smartphone 9, which is implemented bythe far-distance communication circuit 911 illustrated in the FIG. 12,when operating under control of the CPU 901, implements the similar orsubstantially the similar function to that of the data exchange unit 31.

The acceptance unit 92, which is implemented by the touch panel 921,when operating under control of the CPU 901, implements the similar orsubstantially the similar function to that of the acceptance unit 32.

The image and audio processor 93, which is implemented by instructionsof the CPU 901, implements the similar or substantially the similarfunction to that of the image and audio processor 33.

The display control 94, which is implemented by instructions of the CPU901, implements the similar or substantially the similar function tothat of the display control 34.

The determination unit 95, which is implemented by instructions of theCPU 901, implements the similar or substantially the similar function tothat of the determination unit 35.

The generator 96, which is implemented by instructions of the CPU 901,implements the similar or substantially the similar function to that ofthe generator 36.

The communication unit 98, which is implemented by instructions of theCPU 901, implements the similar or substantially the similar function tothat of the communication unit 38.

The data storage/read unit 99, which is implemented by instructions ofthe CPU 901, stores data or information in the memory 9000 and read outdata or information from the memory 9000.

<Operation>

<Participation Process>

Referring to FIG. 18 to FIG. 22, a description is given hereinafter ofoperation according to the embodiment. Firstly, a process ofparticipating in a specific communication session is described withreference to FIG. 18 and FIG. 19. FIG. 18 is a sequence diagramillustrating operation of participating in the specific communicationsession. FIG. 19 is a view illustrating a selection screen for acceptingselection of a desired communication session (virtual conference).

First, the acceptance unit 32 of the videoconference terminal 3 acceptsan instruction to display the selection screen for the communicationsession (virtual conference room), which is input by a user (e.g., theuser A1) at the site A. Then, the display control 34 controls thedisplay 4 to display the selection screen as illustrated in FIG. 19(S21). The selection screen displays selection buttons b1, b2, and b3,which respectively represent virtual conference rooms R1, R2, R3, eachbeing a selection target. Each of the selection buttons b1, b2, and b3is associated with the session ID.

When the user A1 selects a desired selection button (in this example,the selection button b1) on the selection screen, the acceptance unit 32accepts selection of a communication session (S22). Then, the dataexchange unit 31 transmits a request for participating in a virtualconference room to the communication management system 5 (S23). Thisparticipation request includes the session ID identifying thecommunication session for which the selection is accepted at S22, andthe IP address of the videoconference terminal 3 as a request senderterminal. The communication management system 5 receives theparticipation request at the data exchange unit 51.

Next, the data storage/read unit 99 performs a process for enabling thevideoconference terminal 3 to participate in the communication session(S24). More specifically, in the session management DB 5001 (FIG. 16),the data storage/read unit 99 adds the IP address that is received atS23 to an field of the participating terminal IP address in a record ofthe same session ID as the session ID that is received at S23. The dataexchange unit 51 transmits a response to the participation request tothe videoconference terminal 3 (S25). This response to the participationrequest includes the session ID that is received at S23, and a result ofthe participation process. The videoconference terminal 3 receives theresponse to the participation request at the data exchange unit 31. Thefollowing describes a case in which the process for enabling thevideoconference terminal 3 to participate in the communication sessionis successfully completed.

<Management Process of Image Type Information>

Hereinafter, referring to FIG. 20, a description is given of amanagement process of the image type information. FIG. 20 is a sequencediagram illustrating an operation of managing the image typeinformation.

First, when a user (e.g., the user A1) at the site A connects the cradle2 a, on which the image capturing device 1 a is mounted, to thevideoconference terminal 3, using the wired cable such as a USB cable,the storage/read unit 19 a of the image capturing device 1 a reads outthe GUID of the own device (e.g., the image capturing device 1 a) fromthe memory 1000 a. Then, the communication unit 18 a transmits the owndevice's GUID to the communication unit 38 of the videoconferenceterminal 3 (S51). The videoconference terminal 3 receives the GUID ofthe image capturing device 1 a at the communication unit 38.

Next, the determination unit 35 of the videoconference terminal 3determines whether the same vendor ID and product ID as those of theGUID received at S51 are stored in the image capturing device managementDB 3002 (FIG. 15) to determine the image type (S52). More specifically,the determination unit 35 determines that the image capturing device 1 ais an image capturing device that captures a special image (a fullspherical panoramic image, in this disclosure), in a case in which thesame vender ID and product ID are stored in the image capturing devicemanagement DB 3002. By contrast, the determination unit 35 determinesthat the image capturing device 1 a is an image capturing device thatcaptures a general image, in a case in which the same vender ID andproduct ID are not stored in the image capturing device management DB3002.

Next, the data storage/read unit 39 stores, in the image type managementDB 3001 (FIG. 14), the IP address of the own terminal (i.e.,videoconference terminal 3) as the sender terminal, and the image typeinformation as a determination result determined at S52, in associationwith each other (S53). In this state, the image data ID is not yetassociated. Examples of the image type information include the sourcename that is determined according to a predetermined naming rule, andthe image type (general image or special image type).

Then, the data exchange unit 31 transmits a request for adding the imagetype information to the communication management system 5 (S54). Thisrequest for adding image type information includes the IP address of theown terminal as a sender terminal, and the image type information, bothbeing stored at S53 in association with each other. The data exchangeunit 51 of the communication management system 5 receives the requestfor adding the image type information.

Next, the data storage/read unit 59 of the communication managementsystem 5 searches the session management DB 5001 (FIG. 16) using the IPaddress of the sender terminal received at S54 as a search key, to readout the session ID associated with the IP address (S55).

Next, the generator 56 generates a unique image data ID (S56). Then, thedata storage/read unit 59 stores, in the image type management DB 5002(FIG. 17), a new record associating the session ID that is read out atS55, the image data ID generated at S56, the IP address of the senderterminal and the image type information that are received at S54, withone another (S57). The data exchange unit 51 transmits the image data IDgenerated at S56 to the videoconference terminal 3. The videoconferenceterminal 3 receives the image data ID at the data exchange unit 31(S58).

Next, the data storage/read unit 39 of the videoconference terminal 3stores, in the image type management DB 3001 (FIG. 14), the image dataID received at S58, in association with the IP address of the ownterminal (i.e., videoconference terminal 3) as the sender terminal andthe image type information that are stored at S53 (S59).

Further, the data exchange unit 51 of the communication managementsystem 5 transmits a notification of addition of the image typeinformation to the smartphone 9 as other communication terminal (S60).This notification of addition of the image type information includes theimage data ID generated at S56, and the IP address of the own terminal(i.e., videoconference terminal 3) as the sender terminal and the imagetype information that are stored at S53. The smartphone 9 receives thenotification of addition of the image type information at the dataexchange unit 91. The destination to which the data exchange unit 51transmits the notification is other IP address that is associated withthe same session ID as that associated with the IP address of thevideoconference terminal 3 in the session management DB 5001 (FIG. 16).In other words, the destination is other communication terminal that isin the same virtual conference room as the videoconference terminal 3.

Next, the data storage/read unit 99 of the smartphone 9 stores, in theimage type management DB 9001 (FIG. 14), a new record associating theimage data ID, the IP address of the sender terminal, and the image typeinformation, which are received at S60 (S61). In substantially the samemanner, the notification of addition of the image type information istransmitted to the PC 7 as other communication terminal. The PC 7 alsostores, in the image type management DB 7001 (FIG. 14), the image dataID, the IP address of the sender terminal, and the image typeinformation. Through the operation as described heretofore, the sameinformation is shared among the communication terminals in the imagetype management DBs 3001, 7001 and 9001, respectively.

<Image Data Transmission Process>

Hereinafter, referring to FIGS. 21 to 23, a description is given of animage data transmission process in video calling.

FIG. 21 is a sequence diagram illustrating an image data transmissionprocess in video calling.

First, the communication unit 18 a of the image capturing device 1 atransmits image data obtained by capturing an image of a subject orsurroundings to the communication unit 38 of the videoconferenceterminal 3 (S101). In this case, because the image capturing device 1 ais a device that is capable of obtaining two hemispherical images fromwhich a full spherical panoramic image is generated, the image data isconfigured by data of the two hemispherical images as illustrated inFIGS. 3A and 3B. The videoconference terminal 3 receives the image dataat the communication unit 38.

Next, the data exchange unit 31 of the videoconference terminal 3transmits, to the communication management system 5, the image datareceived from the image capturing device 1 a (S102). This transmissionincludes an image data ID for identifying the image data as atransmission target. Thus, the communication management system 5receives the image data and the image data ID at the data exchange unit51.

Next, the data exchange unit 51 of the communication management system 5transmits, to the smartphone 9, the image data received from thevideoconference terminal 3 (S103). This transmission includes the imagedata ID for identifying the image data as a transmission target. Thus,the smartphone 9 receives the image data and the image data ID at thedata exchange unit 51.

Next, the data storage/read unit 99 of the smartphone 9 searches theimage type management DB 9001 (FIG. 14) using the image data ID receivedat S103 as a search key, to read out the image type information (sourcename) associated with the image data ID (S104). When the image typeinformation indicates a special image (full spherical panoramic image,in this disclosure), i.e., when the image type information is“Video_Theta”, the image and audio processor 93 generates a fullspherical panoramic image from the image data received at S103, andfurther generates a predetermined-area image (S105). In this case, theimage and audio processor 93 synthesizes the predetermined-area imagewith an icon 191 (described later) indicating a full spherical panoramicimage, based on the image type information (Video_Theta) indicating aspecial image.

The display control 94 controls the display 917 of the smartphone 9 todisplay the predetermined-area image including the icon 191.

When the image type information indicates a general image, i.e., whenthe image type information is “Video”, the image and audio processor 93does not generate a full spherical panoramic image from the image datareceived at S103. In this case, the display control 94 displays ageneral image that does not contain the icon 191.

Next, referring to FIGS. 22A and 22B, a description is given of a stateof video calling. FIGS. 22A and 22B illustrate example states of videocalling. More specifically, FIG. 22A illustrates a case in which theimage capturing device 1 a is not used, while FIG. 22B illustrates acase in which the image capturing device 1 a is used.

First, as illustrated in FIG. 22A, when the camera 312 (FIG. 10) that isbuilt into the videoconference terminal 3 is used, that is, withoutusing the image capturing device 1 a, the videoconference terminal 3 hasto be placed at the corner of a desk, so that the users A1 to A4 can becaptured with the camera 312, because the angle of view is horizontally125 degrees and vertically 70 degrees. This requires the users A1 to A4to talk while looking in the direction of the videoconference terminal3. Further, because the user A1 to A4 look in the direction of thevideoconference terminal 3, the display 4 has also to be placed near thevideoconference terminal 3. This requires the user A2 and the user A4,who are away from the videoconference terminal 3 (FIG. 10), to talk in arelatively loud voice, because they are away from the microphone 314.Further, it may be difficult for the user A2 and A4 to see contentsdisplayed on the display 4.

By contrast, as illustrated in FIG. 22B, when the image capturing device1 a is used, the videoconference terminal 3 and the display 4 can beplaced relatively at the center of the desk, because the image capturingdevice 1 a is capable of obtaining two hemispherical images from which afull spherical panoramic image is generated. This enables the users A1to A4 to talk in a relatively small voice, because they are close to themicrophone 314. Further, it gets easier for the users A1 to A4 to seecontents displayed on the display 4.

Hereinafter, referring to FIGS. 23A and 23B, a description is given of adisplay example on the display 917 at the site B. FIGS. 23A and 23B areviews, each illustrating a display example on the display 917 at thesite B. FIG. 23A is a view illustrating an example in which image datatransmitted from respective ones of the videoconference terminal 3(image capturing device 1 a) and the image capturing device 1 b aredisplayed as they are, that is, without generating a full sphericalpanoramic image and a predetermined-area image. Further, in the exampleof FIG. 23A, image data transmitted from the PC 7 (image capturingdevice 8) is also displayed as it is. FIG. 23B is a view illustrating anexample in which predetermined-area images are displayed, which aregenerated based on full spherical panoramic images generated from theimage data transmitted from the videoconference terminal 3 (imagecapturing device 1 a) and the image capturing device 1 b, respectively.Further, in the example of FIG. 23B, image data transmitted from the PC7 (image capturing device 8) is displayed as it is. In this example, animage of the site A is displayed in an upper-left display area of thedisplay 917. In an upper-right display area, an image of the site B (ownsite) is displayed. In a lower-left display area, an image of the site Cis displayed. Because in this example, simultaneous video calling isperformed among the three sites, no image is displayed in a lower-rightdisplay area.

When the image data transmitted from respective ones of the imagecapturing device 1 a and the image capturing device 1 b, each beingcapable capturing a full spherical panoramic image, are displayed asthey are, images are displayed as illustrated in FIG. 23A. In otherwords, the site A in the upper left area and the site B in the upperright area are each displayed as the front-side hemispherical image asillustrated in FIG. 3A and the back-side hemispherical image asillustrated in FIG. 3B.

By contrast, when the image and audio processor 93 generates a fullspherical panoramic image from the image data transmitted fromrespective ones of the image capturing device 1 a and the imagecapturing device 1 b, each being capable of obtaining two hemisphericalimages from which a full spherical panoramic image is generated, andfurther generates a predetermined-area image, a predetermined-areaimage, which is a flat image, is displayed as illustrated in FIG. 23B.The general image is displayed in both of FIGS. 23A and 23B because atthe site C, the image capturing device 8 that obtains a general image isprovided.

Further, at the upper left corner of the images of the site A and thesite B, the icon 191 indicating a full spherical panoramic image isdisplayed. In another example, the icon 191 may be displayed at anylocation other than the upper left corner, such as an upper right, lowerleft, or lower right corner of the image. The icon 191 illustrated inFIG. 23B is just one example, and an icon having any suitable shape,size, or color, may be used. Further, in alternative to or in additionto the icon 191, a character string such as “Full spherical image”, or acombination of an icon and characters may be used.

Furthermore, a user is able to change a predetermined area for thepredetermined-area image in the same full spherical panoramic image.More specifically, when the user B1 or the user B2 moves his/her fingeron the touch panel 921 of the smartphone 9, the acceptance unit 92detects the movement of the finger. The display control 94 shifts,rotates, reduces, or enlarges the predetermined-area image based on themovement of the finger detected by the acceptance unit 92. This enablesto shift the predetermined-area image so that the user A3 and the userA4 are displayed, even in a case in which the predetermined-area imagedisplayed according to an initial setting (by default) contains only apart of the users at the site A, that is, the user A1 and the user A2 asillustrated in FIG. 23B.

Further, referring to FIGS. 24A and 24B and FIGS. 25A and 25B, adescription is given of other display examples on the display 917 at thesite B. FIGS. 24A and 24B and FIGS. 25A and 25B are views, eachillustrating a display example on the display 917 at the site B.

As illustrated in FIG. 23B, when only a part of the users at the site A,that is, the user A1 and the user A2, are displayed by default, the userB1 who is viewing this image has difficulty in recognizing whether thisimage is a general image or a special image (full spherical panoramicimage, in this disclosure), unless the user B1 touches thepredetermined-area image to scroll the image. In view of this, when theimage type information that is read out at S104 indicates a specialimage, the display control 94 multiplies, by a given number (e.g., twotimes), an initial setting value (default value) of an angle of view αof the coordinates (x(rH), y(rV), and angle of view α) including anangle of view in a three-dimensional virtual space. Thus, as illustratedin FIG. 24A, a predetermined-area image displaying the site B (own site)in a middle display area and a predetermined-area image displaying thesite A in a bottom display area are displayed widened. In this case,because an image of the site C is a general image, the display control94 does not widen an angle of view. This enables the user B1 or the userB2 at the site B to recognize at a glance that an image capturing devicethat is capable of obtaining a special image is used at the site A. Whenthe angle of view α is widened, even the flat image looks curved becausean original full spherical panoramic image is curved.

Further, switch buttons b1 and b2 are displayed at the lower rightcorners of the display areas of the site B and the site A, respectively.For example, when the user B1 presses (touches) the switch button b1,the acceptance unit 92 accepts an instruction for switch of display. Inresponse to accepting this instruction at the acceptance unit 92, thedisplay control 94 switches the size of full spherical panoramic imageinto the size of general image as illustrated in FIG. 24B. In this case,the display control 94 restores the angle of view α of the virtualcamera IC back to the initial setting value. Thus, an image representinga site such as the site B where the number of users is small is suitablefor viewing when the angle of view α is restored back to an initialsetting value, rather than an extremely curved image displayed bywidening the angle of view α. When the user B1 presses the switch buttonb1 on a screen illustrated in FIG. 24B, the display control 94 switchesthe screen back to that illustrated in FIG. 24A.

Further, for a site such as the site A where the number of users islarge, the display control 94 combines (joining) a plurality ofpredetermined-area images (in this example, two predetermined-areaimages) together, each predetermined-area image having the angle of viewα of initial setting (default) value, as illustrated in FIG. 25A,without widening the angle of view α. In FIG. 25A, an image obtained byjoining a first predetermined-area image containing the user A1 and theuser A2 and a second predetermined-area image containing the user A4 andthe user A3 together side by side is displayed. This enables to displayan image having fewer curves that is suitable for viewing. When theswitch button b2 is pressed on a screen illustrated in FIG. 25A, thedisplay control 94 switches the screen back to the curved image asillustrated in FIG. 24A. In another example, the angle of view α foreach of the first predetermined-area image and the secondpredetermined-area image may be multiplied by 1.1, in place of twotimes, and thereafter the first predetermined-area image and the secondpredetermined-area image are joined together. In still another example,the angle of view for the first predetermined-area image is multipliedby 0.9 while the angle of view for the second predetermined-area imageis multiplied by 1.1, and thereafter the first predetermined-area imageand the second predetermined-area image are joined together.

In addition, in a case in which presentation material is to be displayedin video calling, the display control 94 displays the image of each sitein the vicinity of an area where presentation material is displayed, asillustrated in FIG. 25B.

Although in the above a description is given of processes performed bythe display control 94, the display control 34 and the display control74 are capable of performing the similar or substantially the similarprocesses.

As described heretofore, according to the present embodiment, thecommunication terminal such as the videoconference terminal 3 generatesa full spherical panoramic image, and further generates apredetermined-area image, depending on the image type informationassociated with the image data ID that is transmitted along with imagedata. This prevents the front side hemispherical image and the back sidehemispherical image from being displayed as illustrated in FIG. 23A.

Further, in a case in which the image type information indicates aspecial image (in this disclosure, full spherical panoramic image), thedisplay control 94 widens a predetermined-area image in which the angleof view α (see FIG. 7) of the virtual camera IC in a three-dimensionalvirtual space, as illustrated in FIG. 24A. This enables the user at eachof different sites to recognize that a special image is displayed. Thisprevents the user from misunderstanding that only two users A1 and A2are participating at the site A, although in fact four users A1, A2, A3and A4 are participating, as illustrated in FIG. 23B. In addition, whensix users are participating at the site A, for example, the user B1 orB2 who views a predetermined-area image in which the angle of view α ofthe virtual camera IC is widened as illustrated in FIG. 24A canrecognize at a glance that the image of the site A is different from theimage of the site C. Accordingly, even in a case in which only fourusers A1, A2, A3 and A4 are displayed, the user B1 or B2 can recognizewithout difficulty that there may be other users who are not displayed,because the image of the site A is full spherical panoramic image.

According to embodiments of the present disclosure, users of each ofdifferent sites are able to recognize that a special image is displayed.This enables to prevent the users at different sites frommisunderstanding the number of users who are participating in videocalling.

The above-described embodiments are illustrative and do not limit thepresent disclosure. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present disclosure.

Any one of the above-described operations may be performed in variousother ways, for example, in an order different from the one describedabove.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), DSP (digital signal processor), FPGA (fieldprogrammable gate array) and conventional circuit components arranged toperform the recited functions.

What is claimed is:
 1. A communication terminal comprising: a memory tostore image data identification information identifying image data inassociation with image type information indicating a type of an imagerepresented by the image data; a receiver to receive specific image dataand specific image data identification information identifying thespecific image data from another communication terminal; and circuitryto control a display to display a specific image represented by thespecific image data received by the receiver in a size according to theimage type information associated with the specific image identificationinformation in the memory.
 2. The communication terminal of claim 1,wherein the type of image indicated by the image type informationincludes one of a general image and a special image, and when the imagetype indicated by the image type information associated with thespecific image identification information is a special image, thecircuitry control display of the specific image to have a size largerthan an image displayed when the image type indicated by the image typeinformation is a general image.
 3. The communication terminal of claim2, wherein when the special image is a full spherical panoramic image,the circuitry controls display of a predetermined-area image, which isan image of a predetermined-area of the full spherical panoramic image,to have a size larger than the general image.
 4. The communicationterminal of claim 3, wherein the predetermined-area image is specifiedby a position coordinate including an angle of view of a virtual camerabeing at a viewpoint viewing the full spherical panoramic image that isrepresented as a solid sphere in a three-dimensional virtual space, andthe circuitry widens the angle of view from an angle set by default tocontrol display of the predetermined-area image to have a size largerthan the general image.
 5. The communication terminal of claim 4,wherein the circuitry controls display of the full spherical panoramicimage to switch a size of the full spherical panoramic image to a sizeof the general image.
 6. The communication terminal of claim 5, whereinwhen the circuitry controls display of the full spherical panoramicimage to switch the size of the full spherical panoramic image to a sizeof the general image, the circuitry restores the angle of view of thevirtual camera back to an initial setting value.
 7. The communicationterminal of claim 3, wherein the predetermined-area image is specifiedby a position coordinate including an angle of view of a virtual camerabeing at a viewpoint viewing the full spherical panoramic image that isrepresented as a solid sphere in a three-dimensional virtual space, andthe circuitry joins a plurality of predetermined-area images together,the plurality of predetermined-area images being specified by respectiveones of position coordinates including the angle of views of differentvirtual cameras to control display of the predetermined-area image tohave a size larger than the general image.
 8. The communication terminalof claim 7, wherein the angle of view of each of the different virtualcameras is kept at an initial setting value.
 9. The communicationterminal of claim 1, wherein the circuitry displays, as the image data,material data representing a presentation material in addition to imagedata obtained by capturing by an image capturing device.
 10. A methodfor controlling display of an image, performed by a communicationterminal, the method comprising: storing image data identificationinformation identifying image data in association with image typeinformation indicating a type of an image represented by the image data;receiving specific image data and specific image data identificationinformation identifying the specific image data from anothercommunication terminal; and controlling a display to display a specificimage represented by the specific image data received by the receiver ina size according to the image type information that is stored inassociation with the specific image identification information.
 11. Anon-transitory computer-readable storage medium storing acomputer-executable product that causes a computer to perform a methodfor controlling display of an image, the method comprising: storingimage data identification information identifying image data inassociation with image type information indicating a type of an imagerepresented by the image data; receiving specific image data andspecific image data identification information identifying the specificimage data from another communication terminal; and controlling adisplay to display a specific image represented by the specific imagedata received by the receiver in a size according to the image typeinformation that is stored in association with the specific imageidentification information.